A dual-branch selective attention capsule network for classifying kiwifruit soft rot with hyperspectral images.

Kiwifruit soft rot is highly contagious and causes serious economic loss. Therefore, early detection and elimination of soft rot are important for postharvest treatment and storage of kiwifruit. This study aims to accurately detect kiwifruit soft rot based on hyperspectral images by using a deep learning approach for image classification. A dual-branch selective attention capsule network (DBSACaps) was proposed to improve the classification accuracy. The network uses two branches to separately extract the spectral and spatial features so as to reduce their mutual interference, followed by fusion of the two features through the attention mechanism. Capsule network was used instead of convolutional neural networks to extract the features and complete the classification. Compared with existing methods, the proposed method exhibited the best classification performance on the kiwifruit soft rot dataset, with an overall accuracy of 97.08% and a 97.83% accuracy for soft rot. Our results confirm that potential soft rot of kiwifruit can be detected using hyperspectral images, which may contribute to the construction of smart agriculture.

Effects of phantom microstructure on their optical properties.

Significance: Developing stable, robust, and affordable tissue-mimicking phantoms is a prerequisite for any new clinical application within biomedical optics. To this end, a thorough understanding of the phantom structure and optical properties is paramount. Aim: We characterized the structural and optical properties of PlatSil SiliGlass phantoms using experimental and numerical approaches to examine the effects of phantom microstructure on their overall optical properties. Approach: We employed scanning electron microscope (SEM), hyperspectral imaging (HSI), and spectroscopy in combination with Mie theory modeling and inverse Monte Carlo to investigate the relationship between phantom constituent and overall phantom optical properties. Results: SEM revealed that microspheres had a broad range of sizes with average (13.47±5.98) µm and were also aggregated, which may affect overall optical properties and warrants careful preparation to minimize these effects. Spectroscopy was used to measure pigment and SiliGlass absorption coefficient in the VIS-NIR range. Size distribution was used to calculate scattering coefficients and observe the impact of phantom microstructure on scattering properties. The results were surmised in an inverse problem solution that enabled absolute determination of component volume fractions that agree with values obtained during preparation and explained experimentally observed spectral features. HSI microscopy revealed pronounced single-scattering effects that agree with single-scattering events. Conclusions: We show that knowledge of phantom microstructure enables absolute measurements of phantom constitution without prior calibration. Further, we show a connection across different length scales where knowledge of precise phantom component constitution can help understand macroscopically observable optical properties.

Hyperspectral dark-field microscopy of human breast lumpectomy samples for tumor margin detection in breast-conserving surgery.

Significance: Hyperspectral dark-field microscopy (HSDFM) and data cube analysis algorithms demonstrate successful detection and classification of various tissue types, including carcinoma regions in human post-lumpectomy breast tissues excised during breast-conserving surgeries. Aim: We expand the application of HSDFM to the classification of tissue types and tumor subtypes in pre-histopathology human breast lumpectomy samples. Approach: Breast tissues excised during breast-conserving surgeries were imaged by the HSDFM and analyzed. The performance of the HSDFM is evaluated by comparing the backscattering intensity spectra of polystyrene microbead solutions with the Monte Carlo simulation of the experimental data. For classification algorithms, two analysis approaches, a supervised technique based on the spectral angle mapper (SAM) algorithm and an unsupervised technique based on the K-means algorithm are applied to classify various tissue types including carcinoma subtypes. In the supervised technique, the SAM algorithm with manually extracted endmembers guided by H&E annotations is used as reference spectra, allowing for segmentation maps with classified tissue types including carcinoma subtypes. Results: The manually extracted endmembers of known tissue types and their corresponding threshold spectral correlation angles for classification make a good reference library that validates endmembers computed by the unsupervised K-means algorithm. The unsupervised K-means algorithm, with no a priori information, produces abundance maps with dominant endmembers of various tissue types, including carcinoma subtypes of invasive ductal carcinoma and invasive mucinous carcinoma. The two carcinomas' unique endmembers produced by the two methods agree with each other within <2% residual error margin. Conclusions: Our report demonstrates a robust procedure for the validation of an unsupervised algorithm with the essential set of parameters based on the ground truth, histopathological information. We have demonstrated that a trained library of the histopathology-guided endmembers and associated threshold spectral correlation angles computed against well-defined reference data cubes serve such parameters. Two classification algorithms, supervised and unsupervised algorithms, are employed to identify regions with carcinoma subtypes of invasive ductal carcinoma and invasive mucinous carcinoma present in the tissues. The two carcinomas' unique endmembers used by the two methods agree to <2% residual error margin. This library of high quality and collected under an environment with no ambient background may be instrumental to develop or validate more advanced unsupervised data cube analysis algorithms, such as effective neural networks for efficient subtype classification.

In situ monitoring of the shikimate pathway: a combinatorial approach of Raman reverse stable isotope probing and hyperspectral imaging.

Sensing and visualization of metabolites and metabolic pathways in situ are significant requirements for tracking their spatiotemporal dynamics in a non-destructive manner. The shikimate pathway is an important cellular mechanism that leads to the de novo synthesis of many compounds containing aromatic rings of high importance such as phenylalanine, tyrosine, and tryptophan. In this work, we present a cost-effective and extraction-free method based on the principles of stable isotope-coupled Raman spectroscopy and hyperspectral Raman imaging to monitor and visualize the activity of the shikimate pathway. We also demonstrated the applicability of this approach for nascent aromatic amino acid localization and tracking turnover dynamics in both prokaryotic and eukaryotic model systems. This method can emerge as a promising tool for both qualitative and semi-quantitative in situ metabolomics, contributing to a better understanding of aromatic ring-containing metabolite dynamics across various organisms.

Comparing visible and near infrared 'point' spectroscopy and hyperspectral imaging techniques to visualize the variability of apple firmness.

In this work, visible and near-infrared 'point' (Vis-NIR) spectroscopy and hyperspectral imaging (Vis-NIR-HSI) techniques were applied on three different apple cultivars to compare their firmness prediction performances based on a large intra-variability of individual fruit, and develop rapid and simple models to visualize the variability of apple firmness on three apple cultivars. Apples with high degree of intra-variability can strongly affect the prediction model performances. The apple firmness prediction accuracy can be improved based on the large intra-variability samples with the coefficient variation (CV) values over 10%. The least squares-support vector machine (LS-SVM) models based on Vis-NIR-HSI spectra had better performances for firmness prediction than that of Vis-NIR spectroscopy, with the with the Rc2 over 0.84. Finally, The Vis-NIR-HSI technique combined with least squares-support vector machine (LS-SVM) models were successfully applied to visualize the spatial the variability of apple firmness.

Hyperspectral Proximal Sensing for Estimating Photosynthetic Capacities at Leaf and Canopy Scales.

Agronomists, plant breeders, and plant biologists have been promoting the need to develop high-throughput methods to measure plant traits of interest for decades. Measuring these plant traits or phenotypes is often a bottleneck since skilled personnel, resources, and ample time are required. Additionally, plant phenotypic traits from only a select number of breeding lines or varieties can be quantified because the "gold standard" measurement of a desired trait cannot be completed in a timely manner. As such, numerous approaches have been developed and implemented to better understand the biology and production of crops and ecosystems. In this chapter, we explain one of the recent approaches leveraging hyperspectral measurements to estimate different aspects of photosynthesis. Notably, we outline the use of hyperspectral radiometer and imaging to rapidly estimate two of the rate-limiting steps of photosynthesis: the maximum rate of the carboxylation of Rubisco (Vcmax) and the maximum rate of electron transfer or regeneration of RuBP (Jmax).

Remote Sensing Techniques: Hyperspectral Imaging and Data Analysis.

Hyperspectral imaging is a remote sensing technique that enables remote, noninvasive measurement of plant traits. Here, we outline the procedures for camera setup, scanning, and calibration, along with the acquisition of black and white reference materials, which are the key steps in collecting hyperspectral imagery. We also discuss the development of predictive models such as partial least-squares regression, using both large and small datasets, which are used to predict plant traits from hyperspectral data. To ensure practical applicability, we provide code examples that allow readers to immediately implement these techniques in real-world scenarios. We introduce these topics to beginners in an accessible and understandable manner.

Near-infrared hyperspectral imaging and robust statistics for in vivo non-melanoma skin cancer and actinic keratosis characterisation.

One of the most common forms of cancer in fair skinned populations is Non-Melanoma Skin Cancer (NMSC), which primarily consists of Basal Cell Carcinoma (BCC), and cutaneous Squamous Cell Carcinoma (SCC). Detecting NMSC early can significantly improve treatment outcomes and reduce medical costs. Similarly, Actinic Keratosis (AK) is a common skin condition that, if left untreated, can develop into more serious conditions, such as SCC. Hyperspectral imagery is at the forefront of research to develop non-invasive techniques for the study and characterisation of skin lesions. This study aims to investigate the potential of near-infrared hyperspectral imagery in the study and identification of BCC, SCC and AK samples in comparison with healthy skin. Here we use a pushbroom hyperspectral camera with a spectral range of ≈ 900 to 1600 nm for the study of these lesions. For this purpose, an ad hoc platform was developed to facilitate image acquisition. This study employed robust statistical methods for the identification of an optimal spectral window where the different samples could be differentiated. To examine these datasets, we first tested for the homogeneity of sample distributions. Depending on these results, either traditional or robust descriptive metrics were used. This was then followed by tests concerning the homoscedasticity, and finally multivariate comparisons of sample variance. The analysis revealed that the spectral regions between 900.66-1085.38 nm, 1109.06-1208.53 nm, 1236.95-1322.21 nm, and 1383.79-1454.83 nm showed the highest differences in this regard, with <1% probability of these observations being a Type I statistical error. Our findings demonstrate that hyperspectral imagery in the near-infrared spectrum is a valuable tool for analyzing, diagnosing, and evaluating non-melanoma skin lesions, contributing significantly to skin cancer research.

In-field and non-destructive determination of comprehensive maturity index and maturity stages of Camellia oleifera fruits using a portable hyperspectral imager.

To efficiently detect the maturity stages of Camellia oleifera fruits, this study proposed a non-invasive method based on hyperspectral imaging technology. First, a portable hyperspectral imager was used for the in-field image acquisition of Camellia oleifera fruits at three maturity stages, and ten quality indexes were measured as reference standards. Then, factor analysis was performed to obtain the comprehensive maturity index (CMI) by analyzing the change trends and correlations of different indexes. To reduce the high dimensionality of spectral data, the successive projection algorithm (SPA) was employed to select effective feature wavelengths. The prediction models for CMI, including partial least squares regression (PLSR), support vector regression (SVR), extreme learning machine (ELM), and convolutional neural network regression (CNNR), were constructed based on full spectra and feature wavelengths; for CNNR, only the raw spectra were used as input. The SPA-CNNR model exhibited more promising performance (RP = 0.839, RMSEP = 0.261, and RPD = 1.849). Furthermore, PLS-DA models for maturity discrimination of Camellia oleifera fruits were developed using full wavelength, characteristic wavelengths and their fusion CMI, respectively. The PLS-DA model using the fused dataset achieved the highest maturity classification accuracy, with the best simplified model achieving 88.6 % accuracy in prediction set. This study indicated that a portable hyperspectral imager can be used for in-field determination of the internal quality and maturity stages of Camellia oleifera fruits. It provides strong support for non-destructive quality inspection and timely harvesting of Camellia oleifera fruits in the field.

Exploring the potential of hyperspectral imaging for microbial assessment of meat: A review.

Food safety is always of paramount importance globally due to the devasting social and economic effects of foodborne disease outbreaks. There is a high consumption rate of meat worldwide, making it an essential protein source in the human diet, hence its microbial safety is of great importance. The food industry stakeholders are always in search of methods that ensure safe food whilst maintaining food quality and excellent sensory attributes. Currently, there are several methods used in microbial food analysis, however, these methods are often time-consuming and do not allow real-time analysis. Considering the recent technological breakthroughs in artificial intelligence and machine learning, it raises the question of whether these advancements could be leveraged within the meat industry to improve turnaround time for microbial assessments. Hyperspectral imaging (HSI) is a highly prospective technology worth exploring for microbial analysis. The rapid, non-destructive method has the potential to be integrated into food production systems and allows foodborne pathogen detection in food samples, thus saving time. Although there has been a substantial increase in research on the utilisation of HSI in food applications over the past years, its use in the microbial assessment of meat is not yet optimal. This review aims to provide a basic understanding of the visible-near infrared HSI system, recent applications in the microbial assessment of meat products, challenges, and possible future applications.

Hyperspectral imaging combined with blood oxygen saturation for in vivo analysis of small intestinal necrosis tissue.

Acute mesenteric ischemia (AMI) is a clinically significant vascular and gastrointestinal condition, which is closely related to the blood supply of the small intestine. Unfortunately, it is still challenging to properly discriminate small intestinal tissues with different degrees of ischemia. In this study, hyperspectral imaging (HSI) was used to construct pseudo-color images of oxygen saturation about small intestinal tissues and to discriminate different degrees of ischemia. First, several small intestine tissue models of New Zealand white rabbits were prepared and collected their hyperspectral data. Then, a set of isosbestic points were used to linearly transform the measurement data twice to match the reference spectra of oxyhemoglobin and deoxyhemoglobin, respectively. The oxygen saturation was measured at the characteristic peak band of oxyhemoglobin (560 nm). Ultimately, using the oxygenated hemoglobin reflectance spectrum as the benchmark, we obtained the relative amount of median oxygen saturation in normal tissues was 70.0 %, the IQR was 10.1 %, the relative amount of median oxygen saturation in ischemic tissues was 49.6 %, and the IQR was 14.6 %. The results demonstrate that HSI combined with the oxygen saturation computation method can efficiently differentiate between normal and ischemic regions of the small intestinal tissues. This technique provides a powerful support for internist to discriminate small bowel tissues with different degrees of ischemia, and also provides a new way of thinking for the diagnosis of AMI.

Multiparameter Sensing of Oxygen and pH at Biological Interfaces via Hyperspectral Imaging of Luminescent Sensor Nanoparticles.

Chemical dynamics in biological samples are seldom stand-alone processes but represent the outcome of complicated cascades of interlinked reaction chains. In order to understand these processes and how they correlate, it is important to monitor several parameters simultaneously at high spatial and temporal resolution. Hyperspectral imaging is a promising tool for this, as it provides broad-range spectral information in each pixel, enabling the use of multiple luminescent indicator dyes, while simultaneously providing information on sample structures and optical properties. In this study, we first characterized pH- and O2-sensitive indicator dyes incorporated in different polymer matrices as optical sensor nanoparticles to provide a library for (hyperspectral) chemical imaging. We then demonstrate the successful combination of a pH-sensitive indicator dye (HPTS(DHA)3), an O2-sensitive indicator dye (PtTPTBPF), and two reference dyes (perylene and TFPP), incorporated in polymer nanoparticles for multiparameter chemical imaging of complex natural samples such as green algal biofilms (Chlorella sorokiniana) and seagrass leaves (Zostera marina) with high background fluorescence. We discuss the system-specific challenges and limitations of our approach and further optimization possibilities. Our study illustrates how multiparameter chemical imaging with hyperspectral read-out can now be applied on natural samples, enabling the alignment of several chemical parameters to sample structures.

Dynamics and genetic regulation of leaf nutrient concentration in barley based on hyperspectral imaging and machine learning.

Biofortification, the enrichment of nutrients in crop plants, is of increasing importance to improve human health. The wild barley nested association mapping (NAM) population HEB-25 was developed to improve agronomic traits including nutrient concentration. Here, we evaluated the potential of high-throughput hyperspectral imaging in HEB-25 to predict leaf concentration of 15 mineral nutrients, sampled from two field experiments and four developmental stages. Particularly accurate predictions were obtained by partial least squares regression (PLS) modeling of leaf concentrations for N, P and K reaching coefficients of determination of 0.90, 0.75 and 0.89, respectively. We recognized nutrient-specific patterns of variation of leaf nutrient concentration between developmental stages. A number of quantitative trait loci (QTL) associated with the simultaneous expression of leaf nutrients were detected, indicating their potential co-regulation in barley. For example, the wild barley allele of QTL-4H-1 simultaneously increased leaf concentration of N, P, K and Cu. Similar effects of the same QTL were previously reported for nutrient concentrations in grains, supporting a potential parallel regulation of N, P, K and Cu in leaves and grains of HEB-25. Our study provides a new approach for nutrient assessment in large-scale field experiments to ultimately select genes and genotypes supporting plant biofortification.

Spectral responses at visible and near-infrared wavelengths of soybean plants to fungicides / Respostas espectrais em comprimentos de onda visível e infravermelho-próximo de plantas de soja a fungicidas

The study evaluated the efficacy and soybean spectral responses to fifteen foliar fungicide mixtures labeled to control Asian soybean rust. Canopy level reflectance was measured using a multispectral camera onboard a multirotor drone before and two hours after each spray. The third application of fungicides improved control of soybean rust and increased yield. Nevertheless, up to three consecutive foliar fungicides applications did not affect the reflectance of soybean plants at visible and infrared wavelengths. Thus, drones can be a viable strategy for data acquisition regardless of the application of the fungicides.

Esse estudo avaliou a eficácia e as respostas espectrais de plantas de soja a quinze misturas de fungicidas utilizados no controle da ferrugem asiática da soja (FAS). A refletância do nível do dossel foi medida usando uma câmera multiespectral a bordo de um drone multirotor antes e duas horas após cada pulverização. A terceira aplicação de fungicidas melhorou o controle de FAS e aumentou a produtividade. Porém, três aplicações foliares consecutivas de fungicidas não afetaram a refletância de plantas de soja nos comprimentos de onda visível e infravermelho. Assim, drones podem ser uma estratégia viável para aquisição de dados independentemente da aplicação de fungicidas.

Nondestructive nitrogen content estimation in tomato plant leaves by Vis-NIR hyperspectral imaging and regression data models.

The present study aims to estimate nitrogen (N) content in tomato (Solanum lycopersicum L.) plant leaves using optimal hyperspectral imaging data by means of computational intelligence [artificial neural networks and the differential evolution algorithm (ANN-DE), partial least squares regression (PLSR), and convolutional neural network (CNN) regression] to detect potential plant stress to nutrients at early stages. First, pots containing control and treated tomato plants were prepared; three treatments (categories or classes) consisted in the application of an overdose of 30%, 60%, and 90% nitrogen fertilizer, called N-30%, N-60%, N-90%, respectively. Tomato plant leaves were then randomly picked up before and after the application of nitrogen excess and imaged. Leaf images were captured by a hyperspectral camera, and nitrogen content was measured by laboratory ordinary destructive methods. Two approaches were studied: either using all the spectral data in the visible (Vis) and near infrared (NIR) spectral bands, or selecting only the three most effective wavelengths by an optimization algorithm. Regression coefficients (R) were 0.864±0.027 for ANN-DE, 0.837±0.027 for PLSR, and 0.875±0.026 for CNN in the first approach, over the test set. The second approach used different models for each treatment, achieving R values for all the regression methods above 0.96; however, it needs a previous classification stage of the samples in one of the three nitrogen excess classes under consideration.

Confocal hyperspectral microscopic imager for the detection and classification of individual microalgae.

We propose a confocal hyperspectral microscopic imager (CHMI) that can measure both transmission and fluorescent spectra of individual microalgae, as well as obtain classical transmission images and corresponding fluorescent hyperspectral images with a high signal-to-noise ratio. Thus, the system can realize precise identification, classification, and location of microalgae in a free or symbiosis state. The CHMI works in a staring state, with two imaging modes, a confocal fluorescence hyperspectral imaging (CFHI) mode and a transmission hyperspectral imaging (THI) mode. The imaging modes share the main light path, and thus obtained fluorescence and transmission hyperspectral images have point-to-point correspondence. In the CFHI mode, a confocal technology to eliminate image blurring caused by interference of axial points is included. The CHMI has excellent performance with spectral and spatial resolutions of 3 nm and 2 µm, respectively (using a 10× microscope objective magnification). To demonstrate the capacity and versatility of the CHMI, we report on demonstration experiments on four species of microalgae in free form as well as three species of jellyfish with symbiotic microalgae. In the microalgae species classification experiments, transmission and fluorescence spectra collected by the CHMI were preprocessed using principal component analysis (PCA), and a support vector machine (SVM) model or deep learning was then used for classification. The accuracy of the SVM model and deep learning method to distinguish one species of individual microalgae from another was found to be 96.25% and 98.34%, respectively. Also, the ability of the CHMI to analyze the concentration, species, and distribution differences of symbiotic microalgae in symbionts is furthermore demonstrated.

VNIR-NIR hyperspectral imaging fusion targeting intraoperative brain cancer detection.

Currently, intraoperative guidance tools used for brain tumor resection assistance during surgery have several limitations. Hyperspectral (HS) imaging is arising as a novel imaging technique that could offer new capabilities to delineate brain tumor tissue in surgical-time. However, the HS acquisition systems have some limitations regarding spatial and spectral resolution depending on the spectral range to be captured. Image fusion techniques combine information from different sensors to obtain an HS cube with improved spatial and spectral resolution. This paper describes the contributions to HS image fusion using two push-broom HS cameras, covering the visual and near-infrared (VNIR) [400-1000 nm] and near-infrared (NIR) [900-1700 nm] spectral ranges, which are integrated into an intraoperative HS acquisition system developed to delineate brain tumor tissue during neurosurgical procedures. Both HS images were registered using intensity-based and feature-based techniques with different geometric transformations to perform the HS image fusion, obtaining an HS cube with wide spectral range [435-1638 nm]. Four HS datasets were captured to verify the image registration and the fusion process. Moreover, segmentation and classification methods were evaluated to compare the performance results between the use of the VNIR and NIR data, independently, with respect to the fused data. The results reveal that the proposed methodology for fusing VNIR-NIR data improves the classification results up to 21% of accuracy with respect to the use of each data modality independently, depending on the targeted classification problem.

Lithological information extraction and classification in hyperspectral remote sensing data using Backpropagation Neural Network.

The purposes are to solve the isomorphism encountered while processing hyperspectral remote sensing data and improve the accuracy of hyperspectral remote sensing data in extracting and classifying lithological information. Taking rocks as the research object, Backpropagation Neural Network (BPNN) is introduced. After the hyperspectral image data are normalized, the lithological spectrum and spatial information are the feature extraction targets to construct a deep learning-based lithological information extraction model. The performance of the model is analyzed using specific instance data. Results demonstrate that the overall accuracy and the Kappa coefficient of the lithological information extraction and classification model based on deep learning were 90.58% and 0.8676, respectively. This model can precisely distinguish the properties of rock masses and provide better performance compared with the state of other analysis models. After introducing deep learning, the recognition accuracy and the Kappa coefficient of the proposed BPNN model increased by 8.5% and 0.12, respectively, compared with the traditional BPNN. The proposed extraction and classification model can provide some research values and practical significances for the hyperspectral rock and mineral classification.

Hyperspectral data as a biodiversity screening tool can differentiate among diverse Neotropical fishes.

Hyperspectral data encode information from electromagnetic radiation (i.e., color) of any object in the form of a spectral signature; these data can then be used to distinguish among materials or even map whole landscapes. Although hyperspectral data have been mostly used to study landscape ecology, floral diversity and many other applications in the natural sciences, we propose that spectral signatures can be used for rapid assessment of faunal biodiversity, akin to DNA barcoding and metabarcoding. We demonstrate that spectral signatures of individual, live fish specimens can accurately capture species and clade-level differences in fish coloration, specifically among piranhas and pacus (Family Serrasalmidae), fishes with a long history of taxonomic confusion. We analyzed 47 serrasalmid species and could distinguish spectra among different species and clades, with the method sensitive enough to document changes in fish coloration over ontogeny. Herbivorous pacu spectra were more like one another than they were to piranhas; however, our method also documented interspecific variation in pacus that corresponds to cryptic lineages. While spectra do not serve as an alternative to the collection of curated specimens, hyperspectral data of fishes in the field should help clarify which specimens might be unique or undescribed, complementing existing molecular and morphological techniques.

Hyperspectral image spectral-spatial classification via weighted Laplacian smoothing constraint-based sparse representation.

As a powerful tool in hyperspectral image (HSI) classification, sparse representation has gained much attention in recent years owing to its detailed representation of features. In particular, the results of the joint use of spatial and spectral information has been widely applied to HSI classification. However, dealing with the spatial relationship between pixels is a nontrivial task. This paper proposes a new spatial-spectral combined classification method that considers the boundaries of adjacent features in the HSI. Based on the proposed method, a smoothing-constraint Laplacian vector is constructed, which consists of the interest pixel and its four nearest neighbors through their weighting factor. Then, a novel large-block sparse dictionary is developed for simultaneous orthogonal matching pursuit. Our proposed method can obtain a better accuracy of HSI classification on three real HSI datasets than the existing spectral-spatial HSI classifiers. Finally, the experimental results are presented to verify the effectiveness and superiority of the proposed method.